Phase 9 — Crawl and Index Health, and Enhanced Testing

Markers: ▶ being worked on, ⚑ implemented and waiting to be reviewed, ✓ done, ✓✗ closed by deciding not to do it, ? open decision, unmarked = planned.

1. Crawling using synthetic data. A crawl of any size can be run on one machine without fetching anything from the web. The pages are made up as the crawl asks for them, so a run repeats exactly and costs nobody bandwidth.
   1. ▶ A synthetic crawl tab is added to the crawl options. Manage Crawl, then Edit Crawl Options, gains a tab beside the web crawl one. It keeps the settings that still mean something for a made-up web and replaces the ones that do not.
      1. The settings about reaching the live web are dropped. Repeat type, crawl sleep, robots.txt, restrict sites by url, disallowed sites and seed sites all describe fetching from real hosts. A made-up web has no hosts to be polite to and no addresses to restrict.
      2. The shape and size of the made-up web are chosen. A graph type says how the pages link to one another. A graph size says how many pages the web holds. A third setting says which pages the crawl starts from, taking the place of seed sites.
      3. What a page is and what it says are chosen by chance. One model gives the chance a page is html, pdf or an image. Another gives the words that appear on it. The same seed gives the same web every run, so a result can be compared with the one before it.
      4. The settings suit both a small check and a large load. A small web makes it possible to ask whether every page indexed can be recalled and whether results come back in the order expected. A large one puts the web server, the queue server and the fetchers under load worth measuring.
   2. A synthetic crawl makes its own pages rather than fetching them. Where that tab is the one chosen, the fetcher takes its pages from a function that generates them in place of FetchUrl::getPages. Everything after the fetch runs unchanged, so what is measured is the real crawl.
2. The slowdown over a long crawl is found and fixed. A synthetic crawl is run long enough for the time per page to climb. Where it starts climbing says which stage is at fault, and what that reading names is what gets fixed.
3. An index is never left corrupt by a crawl that stops part way through. Phase 4 added guards that keep a corrupt index from taking the server down. Those are guards rather than prevention. This item is the prevention.
   1. Every point that could leave a half-written entry is named. The walk covers the partition append, the dictionary merge and the postings write. What is looked for is a write that a later read takes as a garbage length or offset.
   2. A write is either finished or undone, never left part way. A write goes to a new name and is put in place at the end, or a small mark says a step was begun. On starting again the mark says whether to finish the step or drop it.
   3. The recovery is shown by stopping a crawl on purpose. A synthetic crawl is driven and the queue server is stopped at chosen points in the write path. The index is read back afterward and must come back clean every time.
4. Other work, grouped by what it touched. Each is a deliverable of its own that came up during the arc. They are grouped by the part of Yioop they concern rather than listed one by one.
   1. A large download no longer holds the web server or its memory. One reader pulling a large video stalled every other request for as long as the download took. A three hundred megabyte video ended the process outright.
      1. A large file is drained by the event loop rather than by the request. A resource request with no byte range is now deferred, and the whole file case names the file instead of reading it out. Measured on a two hundred megabyte file: pages answered in twenty seconds before and under five milliseconds after.
      2. A request keeps its streaming context while it waits. The context saying which protocol a reply goes out on was not saved with the rest of a waiting request. A resumed request therefore gathered its whole body in memory. It is saved and restored with the other request state.
      3. One block of IP addresses keeps one share of the link. Files going to the same block divide one turn of the write loop between them. Several downloads from one place cannot take several shares. Traffic between Yioop machines keeps a whole turn each.